Touch screen and graphical user interface

ABSTRACT

A selective input system and associated method is provided which tracks the motion of a pointing device over a region or area. The pointing device can be a touchpad, a mouse, a pen, or any device capable of providing two or three-dimensional location. The region or area is preferably augmented with a printed or actual keyboard/pad. Alternatively, a representation of the location of the pointing device over a virtual keyboard/pad can be dynamically shown on an associated display. The system identifies selections of items or characters by detecting parameters of motion of the pointing device, such as length of motion, a change in direction, a change in velocity, and or a lack of motion at locations that correspond to features on the keyboard/pad. The input system is preferably coupled to a text disambiguation system such as a T9® or Sloppytype™ system, to improve the accuracy and usability of the input system.

CROSS REFERENCE TO RELATED APPLICATIONS

This Application is a Divisional of U.S. patent application Ser. No.10/881,819, filed 28 Jun. 2004, entitled SELECTIVE INPUT SYSTEM BASED ONTRACKING OF MOTION PARAMETERS OF AN INPUT DEVICE, which is aContinuation-in-Part of U.S. Pat. No. 7,286,115, filed 1 Oct. 2003,entitled DIRECTIONAL INPUT SYSTEM WITH AUTOMATIC CORRECTION, whichclaims priority to U.S. Provisional Patent Application Ser. No.60/461,735, filed 9 Apr. 2003, and claims priority to U.S. ProvisionalPatent Application Ser. No. 60/504,552, filed 19 Sep. 2003.

FIELD OF THE INVENTION

The invention relates to input devices and user interfaces. Moreparticularly, the invention relates to the tracking of the positionand/or motion of an input device and selection of items or characterinput based on the tracking.

BACKGROUND OF THE INVENTION

Input devices often comprise means for pointing or selecting, such as bya stylus, finger, or mouse, whereby a user may interact with a device.The user is often required to interact with a user interface, such as akeypad, touchpad, or touch screen, such as to input a desired character.A user typically maneuvers the pointing or selection device over adesired position over the interface, and then taps or sets the pointingdevice, to activate a chosen region or element, e.g. such as an actualor mapped keypad element or character.

A user is often required to perform a large number of selective pointingactions, which can be difficult to perform, and are prone to error.

Furthermore, the user interfaces for many devices are often small, suchas for small electronic devices, e.g. portable cell phones, personaldigital assistants (PDAs), or other devices often used for business,personal, educational, or recreational purposes. The selective pointingfunctions required to operate such small devices have becomeincreasingly difficult and prone to error, as a user must accurately tapon a very small region within a user interface.

Several structures and methods have been described, to facilitate theentry of information within stylus-based devices.

For example, in a Palm personal digital assistant (PDA), availablethrough Palm Inc., of Milpitas, Calif., a handwriting recognitionsystem, such as Graffiti®, is provided, wherein a user, preferably witha stylus, enters shorthand-style simplified patterns within a designatedentry region of an input screen. Entered motions are analyzed todetermine the entered characters, which are then placed within an“active” or cursor region for the device. For example, for a cursorlocation corresponding to time, e.g. 2:30PM, within a scheduleapplication, a user may enter “Meet with Fred”.

Shumin Zhai and Per-Ola Kristensson, Shorthand Writing on StylusKeyboard, Apr. 5-10, 2003, describe a method for computer-based writing,wherein a shorthand symbol is provided and taught for each word,according to a pattern on a stylus keyboard. A gesture pattern istypically made by tapping the first letter of a word, and gliding thestylus over to subsequent characters in a word. A word is recognized bythe pattern of the gesture over the keyboard.

Jennifer Mankoff and Gregory D. Abowd, Error Correction Techniques,submitted to Interact '99, provides a survey of the “design,implementation, and study of interfaces for correcting error prone inputtechnologies”.

Jennifer Mankoff and Gregory D. Abowd, Cirrin: A Word-Level UnistrokeKeyboard for Pen Input, Proceedings of UIST 1998, Technical note. pp.213-214, describe a structure and method for planar entry of words, witha non-planar motion typically used between words. Keyboard designs aredescribed, in which letters are arranged about the periphery of aneutral area. Each word is begun by starting a stylus path within acentral, i.e. neutral region. A user is then required to trace out apath which crosses, i.e. travels through, selected letters, whileentering the central, neutral region as necessary, between successiveletters.

K. Perlin, Quikwriting: Continuous Stylus-Based Text Entry; presented atACM UIST'98 Conference, describes a shorthand for entering text, whereinthe x,y positions of a stylus on a surface are tracked. The surfaceincludes nine zones, including a central resting zone. A token iscreated whenever the stylus enters or exits any of the zones, and thesequence of tokens is used to determine the entered characters. Thesystem typically requires that the stylus begin each word from a centralresting zone. The system also often requires movement between two zonesfor the determined selection of most characters, while for characterswhich are defined to be “frequent”, the movement from a central restingzone to an outer zone and back to the resting zone can be used.

M. Garrett, D. Ward, I. Murray, P. Cowans, and D. Mackay, Implementationof Dasher, an Information Efficient Input Mechanism, presented at LINUX2003 Conference, Edinburgh, Scotland, describe a text entry system whichuses “a language model to offer predictions to the user withoutconstraining the range of words which can be written”, such as for“providing input on keyboardless devices and for disabled users”. Theinput system presents letters which move across a screen, wherein a usernavigates a cursor into the zones for each letter. Zones for commonletters, based on letters and words already presented, are bigger.

Other work describing text input technologies is provided by P. Isokoskiand R. Raisamo, Device Independent Text Input: A Rationale and anExample, Proceedings of the Working Conference on Advanced VisualInterfaces AVI2000, pages 76-83, Palermo, Italy, 2000; P. Isokoski, TextInput Methods for Eye Trackers Using Off-Screen Targets, In Proceedingsof Eye Tracking Research & Applications Symposium 2000, pages 15-22.ACM, 2000; P. Isokoski, Model for Unistroke Writing Time, CHI Letters:Human Factors in Computing Systems, CHI 2001, 3(1):357-364, 2001; P.Isokoski and M. Käki. Comparison of Two Touchpad-Based Methods forNumeric Entry, CHI Letters: Human Factors in Computing Systems, CHI2002, 4(1):25-32, 2002; P. Isokoski and 1. Scott MacKenzie, Text Entryon Mobile Systems: Directions for the Future, CHI 2001 ExtendedAbstracts, page 495, 2001; P. Isokoski and I. S. MacKenzie; Report onthe CHI2001 Workshop on Text Entry on Mobile Systems, SIGCHI Bulletin,p.14, September/October 2001; P. Isokoski and 1. S. MacKenzie. CombinedModel for Text Entry Rate Development, CHI2003 Extended Abstracts, pp.752-753, 2003; P. Isokoski and R. Raisamo, Architecture for PersonalText Entry Methods, In Closing the Gaps: Software Engineering andHuman-Computer Interaction, pp. 1-8. IFIP, 2003.

While such entry systems provide a means for entering information, therequired shorthand or stylus paths are often complex, and movementsrequired for one character are easily mistaken for different characters.A user is therefore often required to retype one or more characters, ifthe mistakes are even noticed.

It would be advantageous to provide an input system that makes selectionor character input based on determined motions of input device over anarea, i.e. the individual characteristic motions which, as a whole, makeup a pattern. The development of such a user input system wouldconstitute a major technological advance.

It would also be advantageous to provide a user input system, whereinselections of items or characters are determined, i.e. distinguished, bydetecting parameters of motion of an input device, such as length ofmotion, a change in direction, a change in velocity, and/or a pause inmotion, at locations that correspond to features on the keyboard/pad.The development of such a user input system would constitute a majortechnological advance.

As well, it would be advantageous to provide an input system which makesselection or character input based on the motion of input device over anarea, which is coupled to a text disambiguation system such as T9® orSloppyType™ system, to improve the accuracy and usability of the inputsystem. The development of such a user input system would constitute afurther major technological advance.

SUMMARY OF THE INVENTION

A selective input system and associated method are provided, which trackthe motion of an input device over an area. The input device can be atouchpad, a mouse, a pen, or any device capable of providing a location,e.g. such as an x-y location and/or a location based on alternate oradditional dimensions. The area is preferably augmented with a printedor actual keyboard/pad. Alternatively, a representation of the locationof the input device over a virtual keyboard/pad can be dynamically shownon an associated display. The system identifies selections of items orcharacters by detecting parameters of motion of the input device, suchas length of motion, a change in direction, a change in velocity, and ora lack of motion locations that correspond to features on thekeyboard/pad. The input system is preferably coupled to a textdisambiguation system, such as a T9® or SloppyType™ system, to improvethe accuracy and usability of the input system.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic view of a character input system, in whichdiscrete positions of an input device within an input area aredetermined;

FIG. 2 is a detailed schematic view of a character input system based onexemplary movement and time-based tracking of an input device;

FIG. 3 is a schematic view of a touch screen, wherein stylus input andchar/key display preferably share a common interface;

FIG. 4 is a schematic view of an alternate device structure, comprisingone or more unprinted inputs linked to a separate display;

FIG. 5 is a schematic view of a printed entry pad and an output and/oreditor display;

FIG. 6 is a schematic view of a selective input system comprising acircular onscreen input area;

FIG. 7 is a schematic view of a character input system based on thetracking of absolute positions of an input device;

FIG. 8 is a flowchart of an exemplary process for device tracking andcharacter input based on the tracking;

FIG. 9 is a schematic block diagram of a directional input system 160incorporating disambiguation and associated external information;

FIG. 10 is a schematic view of preferred processing, input, and displaysystems associated with an input system based on the tracking ofabsolute positions of an input device;

FIG. 11 is a schematic view of an alternate selective input system basedon the tracking of motion and/or position of a pointing device, whereinfunctions of an input region are changeable based on stylus input and/ordevice controls;

FIG. 12 is a schematic view of an alternate selective input system basedon the tracking of motion and/or position of a pointing device, whereinthe input area is changeable for function and/or appearance; and

FIG. 13 is a perspective view of an alternate selective input systembased on the tracking of motion of an input device through a region orvolume.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

FIG. 1 is a schematic view of a user input system 10, in which discretepositions of a pointing device, i.e. instrument 16 within an input area14 are determined. Devices 12 often comprise means 16 for pointing orselecting, such as by a stylus, finger, or mouse, whereby a user mayinteract with a device 12. The user is often required to interact with auser interface area 14, such as a keypad, touchpad, or touch screen,such as to input a desired character 20.

In the user input system 10 shown in FIG. 1, a user typically maneuversthe pointing or selection device 16 over a desired position 24, e.g. 24a, over the interface area 14, and then taps or sets 19 the pointingdevice 16, such as by contacting a pointing tip or pointer 17 to adesired location 24 within an interface area 14, to activate a chosenregion or element, e.g. such as an actual or mapped keypad element orcharacter 20.

As described above, a user is often required to perform a large numberof selective pointing actions 19, which can be difficult to perform, andare prone to error. Furthermore, the user interfaces 14 for many devices12 are often small, such as for small electronic devices, e.g. portablecell phones, personal digital assistants (PDAs), or other devices oftenused for business, personal, educational, or recreational purposes. Theselective pointing functions 19 required to operate such small deviceshave become increasingly difficult and prone to error, as a user mustaccurately tap 19 on very small regions within a user interface.

FIG. 2 is a detailed schematic view 32 of a selective input system 30based on exemplary movement paths 38 and time-based tracking 34, e.g. 34a-34 n, of a pointing device 16, such as within an input region 14, suchas an area or volume 14. The system 30 and associated process 98 (FIG.8) identifies character selections, by detecting path starts 42,directional changes 46, velocity changes, e.g. motion pauses 48, and/orpath ends 50, at or determined to be near to locations that correspondto features within the input region 14.

As seen in FIG. 2, a path 38 of a device 16 may indicate selectedpositions 34 by one or more techniques, such as by the start 42 of apath 38, a determined loop 44 in the path 38, a direction change 46, avelocity change, e.g. slowing down or pausing in a location 34, anacceleration away from a location 34, or the end 50 of a path 38. One ormore of selected position techniques may be implemented within aselective input system 30, and may also be integrated with the use ofpointing or tapping 19 (FIG. 1), such that a device 12 may be easily andintuitively operated by a user.

For example, causing the pointing device 16 to hover over a character 20for a certain amount of time 36 can be interpreted as a selection of thecharacter 20. Moving the pointing device 16 towards and away from acharacter 20, such as to and from and from a character 20, i.e. changingdirection 46, can be interpreted as a selection 20. Circling 44 acharacter can be interpreted as a selection. A slowing down or pausing48 motion over a character 20 can also be interpreted as selection of acharacter 20.

While the exemplary selective input system 30 shown in FIG. 2 is basedon two-dimensional movement 38 and time-based tracking 34, e.g. 34 a-34n, of a pointing device 16, within an area 14, alternate systemembodiments of the selective input system 30 provide three-dimensionaltracking, such as in relation to an X-axis 18 a, a Y-axis 18 b, and aZ-axis 18 c, or in relation to other coordinate systems 92, e.g. 92 a-92c (FIG. 6).

While the change in direction 46 in FIG. 2 is shown to occur within asmall area, comprising a small radius of curvature 47, a change ofdirection may alternately be determined by other path geometries orcharacteristics, e.g. such as but not limited to a change in directionover a curve having an estimated radius that is less than a thresholdgeometry, a sharp cusp edge in a path, or a comparison of path directionbefore and after a curve or cusp, e.g. such as a change in directiongreater than a threshold angle may be used to signify a selectionlocation 34.

In some system embodiments, circling the same selectable region orcharacter 20 multiple times can be interpreted as selecting thecharacter multiple times. Similarly, movement back and forth over aselectable region or character 20 can be interpreted as selecting thesame character 20 multiple times.

The selective input system 30 may be implemented on a wide variety ofdevices 12, such as but not limited to personal computers, mobiledevices, appliances, controls, and other microprocessor-based devices,such as portable digital assistants, network enabled cell phones, orother devices often used for business, industrial, personal,educational, or recreational purposes.

FIG. 3 is a schematic view 60 of a touchscreen device 12 a, such as apersonal digital assistant (PDA) or a tablet personal computer (PC),where the stylus input and char/key display are effectively the same. Acombined region 62 typically comprises a stroke input region 14 a, aswell as a display region 64, such as to display characters, words, orkeys. The combined region 62 may also preferably provide a messagedisplay region 66. The device may also comprise supplementary controls68, such as for navigation within or between device functions.

FIG. 4 is a schematic view 70 of an alternate device structure 12 b,comprising an unprinted input 14 b, e.g. such as a trackpad 72 a, atablet 72 b, and/or a joystick 72 k, linked, such as through a processor73, to a separate display 76. Selectable movement 38, e.g. 38 a-38 k,through the input device 14 b, e.g. 72 a-72 c, is tracked, and the path38 in the input device 14 b is indicated in a corresponding displayedpath 78, e.g. such as through one or more determined paths 78, i.e. inktrails 78, and/or mouse/crosshair cursors 79. Detected movement paths38, in relation to the input device 14 b, are typically indicated withina char/key area 75, such as in relation to selectable characters, words,items, or keys 20 a-20 n.

As seen in FIG. 4, an optional ink trail 78 provides a means for visualfeedback for hardware configurations 12 in which a char/key area 75(FIG. 4) is displayed on a screen. In some preferred embodiments, theappearance of the displayed ink trail 78 can change, e.g. color orthickness, to indicate that a selection region has been registered,and/or to indicate a level of system confidence that motion in a regionwas properly interpreted as a selection.

The ink trail 78 can additionally be enhanced, such as by a highlight,an illumination, a sparkle, or a blinking selection 20, to indicate oneor more interpreted selections. In some preferred systems 30 b, an inktrail 78 comprises an alternate or complementary audio highlight 81 athrough an audio output circuit 77. For example, the audio highlight 81a can provide audio feedback to a user USR, such as a tone that risesand falls or fades, wherein the pitch or timbre may preferably indicatesystem confidence in tracking 34 or a selection 20. Audio highlighting81 a is often preferable for spatial, i.e. 3-dimensional, systemembodiments 30 g (FIG. 13).

In some system embodiments of the selective input system 30, such as inthe selective input system 30 b shown in FIG. 4, an auxiliary visualfeedback 81 b may preferably be provided, such as to display and/ormagnify a selectable position 20, such as a selectable area or letter20, which is determined to be closest to the current tracked location34. For example, in systems 30 which comprise a printed trackpad 72 a ortouch screen 72 b, where a finger is the primary pointing device 16,and/or in low-light situations, the nearest letter 20 or the immediateselectable area surrounding the current location 34 being tracked may bedisplayed or magnified 81 b, such as in or over the display 74. Thevisual highlight 81 b provides a user USR a visual indication 73 ofwhere the pointing device 16 is currently located 34, i.e. makingcontact, such as to increase selection accuracy for a pointing device16, for example when a finger of a user USR blocks a portion of thekeyboard area from view, or for lighting conditions which disableviewing.

FIG. 5 is a schematic view 80 a of a device 12 c comprising a printedinput area 14 c, such as a trackpad and/or labeled phone keypad, whichprovides a permanent character/key area. The input area 14 c is linkedto a text output display window 182 a, for the display of word choicelists, and/or for a text editing display.

FIG. 6 is a schematic view 80 b of a device 12 d comprising an exemplarynon-rectangular, e.g. circular, onscreen input area 14 d, correspondingto a selective input system 30. The screen includes an on-screenkeyboard 14 d and a text display area 62. In alternate systemembodiments, the selectable characters 20 may be printed around thejoystick device 72 k.

The on-screen keyboard area 14 d can take a variety of shapes, includingbut not limited to circle, square, oval and polygon with any number ofsides. The visual representation is typically, but not limited to, atwo-dimensional plane figure.

The on-screen keyboard 14 d may be enhanced by, or even replaced with, aset of compass point letters, such as ‘A’, ‘H’, ‘N’ and ‘U’. Thesecompass pointer letters can also be placed in an interactivepointer/cursor on screen or even around the input device 14 b, such asaround a joystick device 72 k.

The letters in the on-screen keyboard 14 d can be arranged in any orderor orientation. In the preferred layout as shown in FIG. 6, all letters20 have their bottoms towards the center of the first ring 83 a. In analternative layout, all letters 20 may be upright. In the preferredlayout as shown in FIG. 6, the letters are ordered alphabetically. In analternative layout, the letters may follow the Dvorak order. In the 15preferred layout as shown in FIG. 6, the letters start at the 12 o'clockposition. In an alternative layout, the letters may start at the 9o'clock location. Alternatively, the letters may have a moving startingposition in a rotating keyboard in an embodiment, for example, where theinput device is a type of wheel. In the preferred layout as shown inFIG. 6, the letters are placed clockwise in a first character ring 83 a.In an alternate layout, the letters may be placed counterclockwise. Inthe preferred embodiment as shown in FIG. 6, letters 20 occupy differentamount of radians depending upon their frequency of use in the language,giving more frequent letters a larger target area. In some systemembodiments the sizing of letters 20 can also be dynamic, with letters20 more likely to follow the just registered letter given more area.

Similarly, selectable digits 20, i.e. “0” through “9”, can be arrangedin any order or orientation. For example, selectable digits 20 can belocated adjacent to the series of letters 20 assigned to thecorresponding digit keys on a telephone keypad.

The on-screen keyboard 14 d may include letters of a primary inputlanguage, letters of alternate input languages (and/or accentedletters), digits, and punctuation symbols. The keyboard may also includecharacter components for pictographic languages, diacritics and other“zero-width” characters that attach to preceding characters. Thekeyboard may further include tone marks, bidirectional characters,functions indicated by a word or symbol, and symbolic representation ofa set of characters such as “Smart Punctuation”.

The preferred primary text input keyboard as shown in FIG. 6 includesunaccented letters which form an outer ring 83 a, digits which form aninner ring 83 b, and a symbol or an indicator 85 between the letters “z”and “a”, called “Smart Punctuation”, which intuitively determines whichpunctuation is most appropriate based on the word context.

There may be auditory feedback 81 a and/or visual feedback 81 b on eachjoystick movement or button press. For example, as soon as the joystickdirection is registered, a solid or gradient-fill pie wedge shape couldappear on the keyboard, centered on the current direction of tilt.Further, the width of that pie wedge could narrow in proportion to thetilt of the joystick towards the perimeter. The pie wedge can remainmomentarily after the joystick is returned to its center/restingposition. The pie wedge provides a visual cue that the tilt of thejoystick was registered and reinforces the notion that each actionrepresents a range of possible letters.

FIG. 7 is a schematic view 90 of a selective input system 30 based onexemplary movement 38 and time-based tracking 48, e.g. relative orabsolute, of a pointing device 16. As seen in FIG. 6, movementparameters and/or position parameters are readily made in two or threedimensional systems 92, such as in an orthogonal-axis coordinate system92 a, a cylindrical coordinate system 92 b, or a spherical coordinatesystem 92 k.

SloppyType™ in Selective Input Systems. Several embodiments of theselective input system 30, such as systems 30 a-30 d, as seen in FIG. 3through FIG. 7, preferably comprise enhanced disambiguation, such as butnot limited to SloppyType™ disambiguation.

For example, as shown in FIG. 3 and FIG. 6, the selective input system30 may include a text display area 62, as well as a word choice listregion 64 and/or a message area 66. The exemplary word choice list 64typically comprises a list of words that the system 30 predicts aslikely candidates based on the characters entered by ambiguousdirectional input. For example, the most likely word is a default word.The user can either accept the default word with one action, or selectan alternate word with a combination of actions.

The exact spelling sequence of exact characters coincidentally selectedby the user is also displayed 66, e.g. 66 a (FIG. 6). Preferably, thespelling sequence is displayed in a separate area 66 a, such as above orbelow the word choice list 64. Alternatively, the spelling sequence maybe displayed as an entry in the word choice list 64, typically the firstline or the last line. In FIG. 6, the exact spelling sequence 66 a isdisplayed above the word choice list 64.

The last letter 20 entered may also be indicated or highlighted 81, suchas on the on-screen keyboard and/or in the exact spell sequence, such asbut not limited to font change, color change, reverse video or alternatebackground color, underline, bold face or italics, and outline. Anexample of a visual outline or highlight 81 b can be a box or a circle.

All the words on a word choice list 64, other than the exact spellingsequence at the time when the exact spelling sequence is displayed asthe first or last entry, are ordered by a combination of the shortestcalculated distances between the input entry sequence and each letter 20in each word and the recency of use and/or the frequency of use withinthe given language.

In various embodiments of the selective input system 30, a user canselect a specific word from the word choice list, such as through acharacter/word selection area 64 (FIG. 3), a word choice list orapplication text editor 182 a (FIG. 5), and/or through one or moresupplementary controls 174 a-174 k (FIG. 11). Preferably, the method isconsistent with other applications use of scrolling methods andselection button. The system typically comprises a means of selectingthe exact spelling sequence as well as any predicted words. In onepreferred embodiment, the system comprises a next button and a previousbutton, with which the user can navigate forward and backward throughthe word choice list.

In some system embodiments, an “escape hole” 87 is provided, such aslocated on one or more input rings 83, e.g. 83 a,83 b (FIG. 6), thatallows movement into the word list 64 or to system menus. As well, insome system embodiments, a default/accept character 89 is provided, suchas located on one or more input rings 83, e.g. 83 a,83 b, or elsewhereon the onscreen keyboard 14 c, for accepting a current default word andmoving on, such as to the entry of another word.

Alternatively, the selective input system 30 may include a selectionmode switch button, such as through one or more buttons 71 (FIG. 4) orsupplementary controls 174, e.g. 174 a-174 k (FIG. 11). When a selectionmode switch button 71, 174 is pressed, the system enters a selectionmode and the directional input means can be used to scroll forward andbackward through the word choice list.

In addition, selecting a predicted word using a particular means mayreplace the exact spelling sequence as if the letters of the selectedword had been entered directly by the user, and a new list of predictedwords is generated.

The most likely word is the word added if the user does not try toselect a different word. The default word may be a copy of the exactspelling sequence if the user was accurate. Alternatively, it may be theselected word as described above. In addition, the exact spellingsequence may become the default word if a precision method or mode(described below) is used to explicitly choose at least one letter inthe sequence.

Words that are longer than the number of input device actions registeredin the current entry sequence may be included in the prediction list.Alternately, a further means can be provided to extend a selected wordwith completions. For example, longer words that begin with a selectedword may appear on a pop-up list after a button press or directionalinput, similar to the cascading menus on PC windowing systems.

Once a word is entered, the word is typically displayed in the messagearea 66 a.

Alternatively, the selective input system 30 can be implemented as aninput method editor (IME). In this case, the text entered by the systemgoes into whatever program is actively accepting input from the system.Other applications may be linked to the system, or the system may beincorporated as part of another application. These applications includebut are not limited to: instant messaging, electronic mail, chatprograms, web browsing, communication within a video game, supplyingtext to a video game, as well as word processing.

To enter a text message using some embodiments of the selective inputsystem 30, such as but not limited to system 30 shown in FIG. 7, theuser points the input device 14 in the general direction of the desiredletter, and then continues pointing roughly to each letter in thedesired word. Once all letters have been roughly selected, buttons maypreferably be used to select a specific word from the list of potentialmatches. The selected word goes into the message area 66 a, which may bean appropriate text application such as email or instant message.

In some three dimensional systems 30, true motion 38 in respect to az-axis 18 c is tracked. In alternate embodiments, such as seen in FIG.4, through pressure sensitive input from a trackpad 72 a, a tablet 72 b,or a joystick 72 k, pressure-sensitive input information can preferablybe used to determine motion and position in respect to three dimensions92 (FIG. 7), such as in respect to a Z-axis 18 c.

The selective input system 30 and associated method 98 are not limitedto require unistroke entry, i.e. one single continuous gesture, for aword. The system can piece together any single to multi-regionsequences, and wait for the user to choose the best word choice. Forexample, within an alphanumeric entry area 75, for the entry of the word“hello”, the user can drag the stylus 16 from “h” to “e”, then tap twiceon “I”, and stroke up to “o”.

Some preferred embodiments of the selective input system 30 furthercomprise supplementary input 166 (FIG. 10), such as a printed or actualkeyboard or keypad 166. Alternatively, a representation of the location34 of the pointing device 16 over a virtual keyboard or keypad 166 canbe dynamically shown on an associated display 144 (FIG. 10).

FIG. 8 is a flowchart of an exemplary process 98, implemented on acomputer, such as the device 12 or processor 73 (FIG. 4), 142 (FIG. 9,FIG. 10), for device tracking and character input 118 based on thetracking. The system 30 and associated process 98 identify characterselections, by detecting changes in direction, changes in velocity,and/or pauses, at locations that correspond to features on the keyboardor keypad.

The motion of a pointing device 16 is tracked 100 over an input region14, such that the current position 34 of the device 16 is determined 102at subsequent times 36, thereby defining a device path 38. Once thelocation 34 and associated time 36 is determined, the location 34 andassociated time 36 are compared 104 to path data 38. At decision step106, the process determines if the current location 34 and associatedtime 36 meet a threshold of a selectable position or character 20, e.g.such as if the pointing device 16 has changed in direction, changed invelocity, or stopped at a location that corresponds to a feature 20within the area 14, such as corresponding to a the keyboard or keypadelement 20. If the threshold decision is negative 108, the processreturns 60 and continues to track 100 the motion.

While the exemplary process 98 describes a comparison between a singlethe location 34 and associated time 36 to the path, one or more pointscan be analyzed, e.g. such as the current location 34 and the last threeto five locations 34, to determine if a discernable selective motion hasbeen made by the user.

If the threshold decision is positive 112, the process decides 114 ifthe selected position 34 adequately indicates a selected item orcharacter 20, e.g. such as if the identified position 34 is locatedwithin or sufficiently near the bounds of a selectable item or character20. If a selectable item or character 20 is indicated 116, the selecteditem 20 is entered 118, and the process returns 110 and continues totrack 100 the motion. If a selectable item or character 20 is not 120sufficiently indicated, or if the system 30 determines 130 that analternate selection 130 may be a valid or more valid choice, someembodiments of the process 98 disambiguate 122 the selection 34,36 ifpossible 124, and return to track 100 the motion. If an attempt todisambiguate 122 is not successful 126, the system 30, 98 may return totrack 100 the motion, such as by passing over the position, entering ablank character, or prompting the user to correct or reenter theintended selection 20, either by the pointing device 16, or throughsupplementary input 166 (FIG. 10).

The disambiguation process 122 may comprise a determination of theclosest selectable character or item 20, a determination of a group ofselections 20, e.g. a word, or a determination one or more likelyselections 20, by which a user can either choose from the determinedlikely selections 20, or may otherwise enter a corrected selection 20.The disambiguation process 122 may alternately comprise a selection of aposition or vicinity in the region of multiple characters 20, as opposedto focusing on the selection of a single character 20.

In some embodiments of the input system 30 and associated process 98,the disambiguation function 122 comprises a text disambiguation system122, such as a T9® or Sloppytype™ disambiguation system 122, to improvethe accuracy and usability of the input system 30.

Details regarding disambiguation systems and processes 122 are seen inU.S. Pat. No. 5,818,437, entitled REDUCED KEYBOARD DISAMBIGUATINGCOMPUTER; U.S. application Ser. No. 10/677,890, filed 1 Oct. 2003,entitled DIRECTIONAL INPUT SYSTEM WITH AUTOMATIC CORRECTION; U.S.application Ser. No. 09/580,319, filed 26 May 2000, entitled “KEYBOARDSYSTEM WITH AUTOMATIC CORRECTION”; and U.S. Provisional Application60/461,735, filed 9 Apr. 2003, entitled “DIRECTIONAL SLOPPY TYPE”, whichare incorporated herein by reference.

FIG. 9 is a block schematic diagram 140 illustrating an exemplaryselective input system 30 which comprises disambiguation functionality,according to a preferred embodiment of this invention. The selectiveinput system 30 shown in FIG. 9 includes an analog input device 14, e.g.such as a joystick 72 k (FIG. 4), preferably also comprising one or morebuttons 71 (FIG. 4), an external information module 152 which typicallystores a collection of linguistic objects, e.g. words and/or phrases, adisplay device 144 having a text display area, and a processor 142. Theprocessor 142, which connects the other components together, furtherincludes an object search engine 145, a motion and position calculationmodule 147 for calculating distance values, a word and phrase(linguistic object) module 148 for evaluating and ordering words, and aselection component 150. The system 30 may further comprise an optionalon-screen representation of a keyboard 75 (FIG. 4) viewable through thedisplay device 144.

As described above, some preferred system embodiments 30 comprise textdisambiguation functionality, such as to disambiguate the intendedselection 20 of a device, or to provide a user with possible selectionchoices 20 for one or more selectable characters 20 that are determinedto be possible selections, or with word or phrase choices 148 that maypotentially be represented by the selection sequence.

For example, on a standard QWERTY keyboard 166 (FIG. 10), selectablebuttons 20 for the letters “R”, “T”, “F”, and “G” are located relativelyclose. For a determined position 24,34 of a pointing device 16 that liesclose to the adjoining region of the letters “R”, “T”, “F”, and “G”,e.g. location 24 m (FIG. 1), a text disambiguation module 72 maydetermine the likely choices “R”, “T”, “F”, and “G”, such as within thedisplay 144, whereby a user may readily choose the listed word thatincludes the intended selection 20 in the proper position in thecharacter sequence.

FIG. 10 is a schematic view 160 of preferred processing 142, input 146,and display 162 systems associated with an input system 30 d based onthe tracking of absolute positions of a pointing device 16.

The supplementary input 166 typically comprises a printed or actualkeyboard or keypad 166. Alternatively, a representation of the location24,34 of the pointing device 16 over a virtual keyboard or keypad 166can be dynamically shown on an associated display 144.

As seen in FIG. 10, a display 144 may be used to display 168 one or moredetermined selected characters 20, e.g. the misspelled word “Gitaffe”,wherein the exemplary determined position 34 of a pointing device 16 forthe third letter 170 is “t”. In some system embodiments 30, thedetermined characters 20 are displayed for a user, such that a user mayedit one or more selections 20. For example, as seen in FIG. 10, a usermay change the third letter “t” to an “r”, to produce a corrected wordgroup “giraffe” 164, such as through selection of alternate words from aword list, or through cursor selection 170 of one or more letters orcharacters, and entry of a desired letter or character 164, typicallythrough reentry of the pointing device 16, or through the secondaryinput 166.

The input device 14,72 serves as a selection input device, whichprovides a possibility of directional input with a sufficient precision,preferably 10 degrees or more precise. It may preferable that thedefault position of the cursor 79 (FIG. 4), if it is shown, is within anactive region within a viewable display 144, such as at the center of anonscreen region 75. It is possible to use a joystick device to navigatein two dimensions an on-screen “QWERTY” or “ABC” keyboard, either in thestandard rectangular form or in a circular layout. It is also possibleto navigate through multiple concentric rings of characters.

Although an analog joystick 72 k is described as the selection device 14in the selection system 160 shown in FIG. 10, any input device 14 thatprovides the possibility of directional input with a sufficientprecision can be used. For examples: omni-directional rocker switch,thumbstick, e.g. IBM TrackPoint™, touchpad, touchscreen, touchscreen andstylus combination, trackball, eye tracking device, trapped-disk slidingswitch, steering wheel, Apple iPod™ Navigation Wheel, or Sony's Jog-dialand data glove, e.g. old Nintendo Game Glove, can be used asalternatives.

The input system 30 shown in FIG. 10 provides a method for preciselychoosing the letters of a word. The method is useful for enteringuncommon names and any word that is not part of the standard languagecurrently active. The method can also be used to change the lastcharacter entered by stepping between characters adjacent to the lastcharacter entered. To step between characters adjacent to the lastcharacter entered, supplementary input 166, such as a forward buttonand/or a backward button may be used. Once the character 170 entered hasbeen changed 164, the word choice list refreshes to reflect the changesin the predicted words. Alternatively, the system may be switched to aprecision mode and the directional input means may be used to cyclethrough letters. For example, in a joystick configuration 72 k (FIG. 6),to switch to the precision mode, the system may choose to use the degreeof joystick tilt from the center. Once the tilt exceeds a preconfiguredlimit, the system 30 switches to the precision mode. Alternatively, thesystem 30 may use the time interval that the joystick dwells at theperimeter. Once the time interval reaches a preconfigured limit, thesystem switches to the precision mode and notifies the user through avisual cue or a tone. The system may also include a button for switchingto precision mode.

For example, as seen in FIG. 10, a user may change the third letter “t”to an “r”, to produce a corrected word group “giraffe” 164, such asthrough selection of alternate words from a word list, or through cursorselection 170 of one or more letters or characters, and entry of adesired letter or character 164, typically through reentry of thepointing device 16, or through the secondary input 166.

The linguistic objects that are stored in the information module 152(FIG. 9) typically comprise but are not limited to: words, phrases,abbreviations, chat slang, emoticons, user IDs, URLs, and/or non-English(such as Chinese or Japanese) characters. Although words are used in thepreferred embodiments, any other linguistic objects are equallyapplicable. Similarly, although the term “letter” or “character” is usedin the preferred embodiment, other sub-word components from Non-Englishlanguages, e.g. strokes, radicals/components, jamos, kana, pluspunctuation symbols and digits, are equally applicable.

The list of predicted or alternate words is typically ordered inaccordance with a linguistic model, which may include one or more of:frequency of occurrence of a word in formal or conversational writtentext; frequency of occurrence of a word when following a preceding wordor words; proper or common grammar of the surrounding sentence;application context of current word entry; and recency of use orrepeated use of the word by the user or within an application program.

One or more techniques can be implemented within a disambiguationprocess 122 (FIG. 8). In some situations, even if one or more charactersare clearly indicated and selected, the disambiguation process 122 mayreject the word, e.g. such as characters within a misspelled word, andoffer a choice of correctly spelled alternate words, or mayautomatically replace the word, e.g. such as for commonly mistyped wordsor transposed letters. For example, in a situation in which a user USRhas clearly entered an “S”, the disambiguation process may suggest an“A” or a “D”, such as for choices of one or more neighboring selectablecharacters 20 in a QWERTY keyboard, which may be determined to belogical. Therefore, even if a user USR precisely enters or indicates aselectable position 20, the disambiguation process 122 may provide oneor more determined optional choices, e.g. by presenting the user with adisplay note, such as “Is this alternative choice what you meant toenter?”.

As well, the disambiguation process 122 may determine a selection motionat a position which does not clearly indicate a selected position 20within the input region 14.

While the disambiguation process 122 is readily adapted to provide spellchecking, the process 122 can also provide other disambiguation. Forexample, after entering a complete word or phrase, i.e. following a path38 that returns multiple character candidates 20 at each spot 34 alongthe path 38, individual “selected positions 20 can be disambiguated withmultiple mechanisms, wherein the mechanism typically uses context, suchas the current input or display language, adjacent characters or words,previously used words or phrases, and/or known words or phrases.

Alternate System Embodiments. FIG. 11 is a schematic view 171 of analternate selective input system 30 e based on the tracking of absolutepositions of a pointing device 16, wherein functional sets 176, e.g. 176a,176 b, of selectable characters 20 of an input region 14 arechangeable, such as based on stylus input 172 and/or device controls 174a-174 k. For example, the selectable characters 20 may readily bechanged in function to alternate characters 20, and may also beassociated with different display characters. For example, stylus input172 and/or device controls 174 a-174 k may be used as a shift, option,or control keys, whereby selectable characters are replaced withalternate characters.

In alternate system embodiments 30 e, a user does not have to explicitlyselect an input method, i.e. a set 176 of selectable characters. Forexample, by simply using the provided interface, such as through screenentry or by pressing one or more keys, the system may automaticallyswitch or adjust to an alternate input set 176.

In alternate embodiments of the selective input system 30, whereincharacters or locations are selected, the system 30 preferably providesmeans for successive or repeated entry of one or more selectable items,characters, and/or locations, such as through action of a pointingdevice 16, e.g. a loop 44 on or near a selectable item 20, followed by atap 19 in the same region. As well, supplementary input, such as but notlimited to stylus input 172 and/or device controls 174 a-174 k, can beused for successive or repeated entry of one or more selectable items,characters, and/or locations.

In alternate embodiments of the selective input system 30, entirekeyboard arrangements are readily changed, such as to be suitable for anintended function, or based upon the preference of a user. For example,the design of a QWERTY keyboard set 176, can be changed to a DVORAKkeyboard set 176, or a phone directory interface set 176 can be changedto a telephone keypad interface set 176. Similarly, all or part of theinput screen area 14 may be used in combination with shorthand orcharacter recognition type entry, e.g. Graffiti®.

FIG. 12 shows a schematic view 180 of an alternate selective inputsystem 30 f, based on the tracking of absolute positions of a pointingdevice 16, wherein the input region 14 is changeable for function and/orappearance. For example, the device 12 is readily adapted to provide aplurality of different functions 182 a-182 j, through which theselective input system 30 e tracks the motion parameters of a pointingdevice 16, e.g. such as but not limited to a text entry 182 a, spellchecking 182 b, an alternate keyboard 182 c, a schedule interface 182 d,a to-do list 182 e, or other graphic interfaces 182 f, 182 j.

As seen in to-do list interface 182 e, a user may readily input desiredpriorities 186 on a group of tasks 184, such that the tasks 184 arereadily rearranged, based on the tracked path 38. As seen in graphicarea interface 182 f, a user may readily input a desired travelitinerary 188, based on tracked travel path 186 on a map 190. Similarly,a graphic area interface 112 j can display a game interface 192, wherebythe tracking of motions and/or positions of a pointing device 16determines game actions decisions and/or outcomes 194, such as success194 a and/or accumulated points 194 b.

FIG. 13 is a perspective view 200 of an alternate selective input system30 g based on the tracking of motion 38 of an input device through aregion or volume 14. The selective input system 30 f and associatedmethod 98 are readily adapted to provide an input system based uponmotion of an actual device 12, such as to detect two orthree-dimensional motion in a variety of coordinate systems 92, e.g. 92a-92 c.

In the exemplary system 30 f shown in FIG. 13, a user USR controllablymoves an input device 12, such as but not limited to a stylus, PDA, cellphone, laser pointer, light pen, bare hand HD, a glove 204, or finger12. For example, in a gaming system, a glove can be used for motiontracking. As well, while the exemplary system 30 f shown in FIG. 13shows the possible tracking of a bare hand HD or a gloved hand 204,other portions of the body of a person may be tracked, such as but notlimited to a foot, a head, or other appendages. Similarly, while theexemplary system 30 f shown in FIG. 13 shows the possible tracking of aglove 204, other accessories, tools, or articles of clothing 204 mayalternately be used as a pointing device 12 for motion tracking, such asbut not limited to a ring, a bracelet, a hat, a toy, or a wand. Forexample, the motion of a pointing tip of a play sword or wand 12 can betracked, which can additionally be displayed on a projection screen 202,such as in context with a game scene.

Upon detection of relative motion and path 38 and/or subsequentpositions 34, the enhanced device 12 is readily used for a wide varietyof applications, such as but not limited writing, game input, and/orcontrol, either within the device itself 12, or in relation to otherdevices 202.

In some alternate selective input systems 30 g, the enhanced inputdevice comprises accelerometers and/or a detection field for detectionof relative motion and path 38 and/or subsequent positions 34. In otheralternate selective input systems 30 g, emitted light from laser pointer12 is projected in relation to a projection screen 202, such as toprovide a two-dimensional input. In yet another alternate selectiveinput system 30 g, emitted light from a light pen is detected on scanlines of a video screen 202, such as to provide an alternatetwo-dimensional input. In another alternate input system 30 g, themotion of a user, e.g. a user's hand or finger, is tracked, such as bybut not limited to a camera, radar, or lidar, such that no device needsto be held by the user USR.

In the exemplary system 30 g shown in FIG. 13, a user USR can cursivelywrite a note, the motion of which is detected, calculated as an entry,disambiguated as necessary, and stored on the device 12. In a similarembodiment, a user can cursively enter a phone number or quick dialentry on an enhanced phone, receive acoustic feedback, such as a messageof the intended number, e.g. “The phone number you entered is555-555-1234”, and have the number dialed automatically. In an alternateembodiment, a user USR can cursively enter a desired cable channel,wherein the motion of which is detected, calculated as an entry,disambiguated as necessary, and sent to an external appliance orcontroller 202.

In the selective input system 30 g, the motion of input device 12 istracked within the region or volume 14, while the user USR preferablygets visual feedback for the current “location” of the enhanced device12, relative to the specific region or volume 14, to provide a pluralityof different functions 182 a-182 j (FIG. 12), e.g. such as but notlimited to a text entry 182 a, spell checking 182 b, an alternatekeyboard 182 c, a schedule interface 182 d, a to-do list 182 e, or othergraphic interfaces 182 f,182 j.

In some preferred embodiments of the selective input system 30 g, thedisplay 202 is separate from the input device 12, such as TV screen 202,e.g. for gaming or channel switching applications, or for a heads-updisplay 202.

System Advantages. The selective input system 30 and associated method98 provide significant advantages over existing character input systems,such that a user may quickly and intuitively enter one or moreselections 20, even within a small interface area 14. In contrast tosystems in which a user must repeatedly and accurately point or tapselections 20, the selective input system 30 and associated method 98allow a much broader use of a pointing device 16, wherein the device 16can be used in a variety of planar or non-planar movements to enter oneor more selectable items or characters 20.

While the selective input system 30 and associated method 98 can be usedin combination with alternate methods of user input, e.g. stylus-basedshorthand or handwriting recognition systems, the selective input system30 and associated method 98 provide an intuitive entry system that canbe used in many input environments. For example, the selective inputmethod 48 can similarly be implemented on a planar keyboard, a keypad,or a telephone interface.

Although the selective input system and methods of use are describedherein in connection with personal computers, mobile devices,appliances, controls, and other microprocessor-based devices, such asportable digital assistants or network enabled cell phones, theapparatus and techniques can be implemented for a wide variety ofelectronic devices and systems, or any combination thereof, as desired.

As well, while the selective input system and methods of use aredescribed herein in connection with interaction between atwo-dimensional input devices and systems, the character input systemand methods of use can readily be implemented for selection within otherdimensional systems, such as for one-dimensional slide controls,three-dimensional location or selection, or any combination thereof, asdesired.

Accordingly, although the invention has been described in detail withreference to a particular preferred embodiments, persons possessingordinary skill in the art to which this invention pertains willappreciate that various modifications and enhancements may be madewithout departing from the spirit and scope of the invention and theclaims that follow.

1. A selective input system for a portable device, comprising: a touchscreen for tracking X-Y motion that corresponds to interaction of a user with said touchscreen when said user performs one or more pointing actions by moving one or more fingers over an input area of said touchscreen; a motion and position calculation module in communication with said touchscreen for receiving information therefrom corresponding to said user pointing actions and for identifying all of input data and device function and control commands based upon said information, said motion and position calculation module determining pointing actions embodied in specific X-Y motions of one or more of said user's fingers comprising length of motion, change in direction, change in velocity, lack of motion, path start and end points, pointing, tapping, looping, and time-based motions and/or sequences of motions at one or more locations, simultaneously or sequentially, relative to said input area; and a processor for receiving said input data and device function and control commands from said motion and position calculation module and form operating said portable device in accordance therewith.
 2. The system of claim 1, wherein a threshold of change in direction comprises any of a curve having an estimated radius that is less than a threshold geometry, a comparison of the device path direction before and after a curve, a sharp cusp edge in the device path, and a comparison of path direction before and after a cusp.
 3. The system of claim 1, wherein the said motion is limited to motion of one or more user fingers on a planar surface of the region.
 4. The system of claim 1, wherein the input area comprises a region in any of a portable digital assistant, a portable telephone, a mobile telephone, a portable computer, and a portable game device.
 5. The system of claim 1, further comprising: means for disambiguating at least one determined position which corresponds to an alternate selection.
 6. The system of claim 5, wherein the disambiguation means comprises: a display of the alternate selection; and a selectable choice by the user between the determined position and the alternate selection.
 7. The system of claim 1, further comprising: means for providing feedback to a user.
 8. The system of claim 7, wherein the feedback comprises any of visual feedback and audio feedback.
 9. The system of claim 8, wherein the visual feedback comprises any of an ink trail corresponding to the determined device path, a font change, a color change, a reverse video color, an alternate background color, an underline, bold face text, italic text, and a text outline,
 10. The system of claim 8, wherein the audio feedback comprises any of a tone indicating system confidence in any of tracking and selection, a sound indicating any of a selection and an entry, and an acoustic message.
 11. The system of claim 1, further comprising: logic for determining a distance between a current location of the user's one or more fingers and selectable positions within the touchscreen; logic for determining a selectable position which is closest to the current location of the user's one or more fingers; and a highlight corresponding to the determined closest selectable position.
 12. The system of claim 11, wherein the highlight comprises any of a display and a magnification of the determined closest selectable position.
 13. The system of claim 1, further comprising: means for disambiguating at least one subsequent position based upon a contextual comparison between the subsequent position and stored information.
 14. The system of claim 13, wherein the stored information comprises any of language, characters, and words.
 15. The system of claim 13, wherein the disambiguation means comprises: a spell check function for one or more selected positions; and a choice for the user of results of the spell check function.
 16. The system of claim 1, further comprising: a disambiguation of at least one subsequent position which does not clearly indicate a selected position within the touchscreen.
 17. The system of claim 16, wherein the disambiguation comprises: a display of at least one selectable position which may correspond to the subsequent position that does not clearly indicate a selected position within the touchscreen; and a selectable choice for the user of the at least one selectable position.
 18. The system of claim 1, wherein the selected positions correspond to any of characters, menu selections, and functions.
 19. The system of claim 1, further comprising: a plurality of interfaces comprising a plurality of selected position within the touchscreen; and means for switching between the interfaces.
 20. The system of claim 19, wherein the means for switching between the interfaces comprises a selection by a user pointing action.
 21. The system of claim 19, wherein the means for switching between the interfaces comprises a supplementary control.
 22. The system of claim 1, further comprising: logic for predicting a default selection; and means for accepting the predicted default selection.
 23. The system of claim 22, wherein the predicted default selection comprises a word.
 24. The system of claim 22, wherein the accepting means comprises a user pointing action comprising any of a directional input, a button selection, a menu selection, a list selection, and a functional input. 